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ABSTRACT : 

PROBLEM TO BE SOLVED: To provide a single CCD color image 
pickup element with 

high image quality by arranging plural color mosaic color 
filters to a position 

corresponding to a picked-up picture element at random 
color arrangement and 

interpolating a missing picture element for each color so 
as to generate all 
picture elements. 

SOLUTION: An image pickup element 100 consisting of a CCD 
image sensor whose 

image pickup face has a mosaic color filter converts a 
received object image 

into an electric image signal under the drive control of an 
image pickup 

element driver 200. A memory 400 stores image information 
converted into a 

digital signal by an A/D converter 300 under the recording 
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control of a memory 

controller 500. A processor 600 applies interpolation 
processing to each of R, 

G, B primary colors with respect to an object image 
recorded in the memory 400 

and generates all picture elements and stores them to a 
color dependent area of 

the memory 400. Then the picture element position of the 
same color filter is 

arranged at random spatially and the arrangement 
information is stored in a ROM 

700 and the processor 600 references the information for 

the interpolation 

processing. 
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Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1 .This document has been translated by computer. So the translation may not reflect the original precisely. 

2 **** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



MEANS 



[Means for Solving the Problem] In order to attain the above-mentioned purpose, the color picture image pck-up 
equipment concerning the 1st invention The image pck-up element which has arranged spatially at random the mosa 
color filter which has two or more colors so that the filter position of each color may correspond to an image pck-up 
pixel, A storing means to store the filter color arrangement information on the aforementioned mosaic color filter, A 
interpolation means to interpolate the image information for every color including lack by interpolation processing i 
the state where there is no lack pixel is provided using the image information outputted from the aforementioned 
image pck-up element, and the filter color arrangement information within the aforementioned storing means. 
[0008] Moreover, the color-picture image pck-up element concerning the 2nd invention provides the image pck-up 
element which has arranged spatially at random the mosaic color filter which has two or more colors by the batch 
interpolated so that the filter position of each color may correspond to an image pck-up pixel, and a storing means 
store the filter color arrangement information on the aforementioned mosaic color filter, in the color-picture image 
pck-up element in the color-picture image pck-up equipment which interpolates the picturized color-picture 
information by the interpolation means, and restores the original picture. 

[0009] Moreover, the color picture image pck-up equipment concerning the 3rd invention The image pck-up elemen 
which has arranged spatially at random the mosaic color filter which has two or more colors so that the filter positio 
of each color may correspond to an image pck-up pixel, A storing means to store the filter color arrangement 
information on the aforementioned mosaic color filter, A storage means to memorize the picture of two or more 
sheets which is outputted by the aforementioned image pck-up element and continues about a time-axis, A position 
gap detection means to detect the same photographic subject's relative location mutually among the pictures of two 
more sheets memorized by this storage means, The relative location detected by this position gap detection means is 
used, the picture of two or more sheets memorized by the aforementioned storage means — the above — with a pictu 
composition means to compound so that the same photographic subject may lap A interpolation means to restore the 
image information for every color including lack to the state where there is no lack pixel, by interpolation processin 
is provided using the image information compounded by this picture composition means and the color arrangement 
information within the aforementioned storing means. 

[0010] That is, the color picture image pck-up equipment concerning the 1st invention stores the filter color 
arrangement information on the aforementioned mosaic color filter in the storing means while arranging spatially at 
random the mosaic color filter which has two or more colors so that the filter position of each color may correspond 
to an image pck-up pixel. And the image information for every color including lack is interpolated by interpolation 
processing in the state where there is no lack pixel, using the image information outputted from the image pck-up 
element, and the filter color arrangement information within the aforementioned storing means. 
[001 1] Moreover, the color-picture image pck-up element concerning the 2nd invention stores the filter color 
arrangement information on the aforementioned mosaic color filter in a storing means in the color-picture image pck 
up element in the color-picture image pck-up equipment which interpolates the picturized color-picture information 
by the interpolation means, and restores the original picture while arranging spatially at random the mosaic color fil 
which has two or more colors by the batch interpolated so that the filter position of each color may correspond to an 
image pck-up pixel. 

[0012] Moreover, the color picture image pck-up equipment concerning the 3rd invention stores the filter color 
arrangement information on the aforementioned mosaic color filter in the storing means while arranging spatially at 
random the mosaic color filter which has two or more colors so that the filter position of each color may correspond 
to an image pck-up pixel, next, the picture of two or more sheets which was outputted by the image pck-up element 
detected the same photographic subject's relative location by the position gap detection means mutually among the 
pictures of two or more sheets memorized by storage means memorized the picture of two or more sheets which 
continues about a time-axis, and was memorized by the aforementioned storage means using this detected relative 



location - the above ~ it compounds by the picture composition means so that the same photographic subject may 1 
And interpolation processing restores the image information for every color including lack to the state where there i 
no lack pixel, using the image information compounded by the aforementioned picture composition means and the 
color arrangement information within the aforementioned storing means. 
[0013] 

[Embodiments of the Invention] With reference to a drawing, the operation form of this invention is explained in 
detail below. First, the 1st operation form of this invention is explained. The whole 1st operation form block diagram 
of this invention is shown in drawing 1 . In the composition of drawing 1 , the image pck-up element 100 which 
consists of CCD series with which the image pck-up side was equipped with the mosaic color filter is changed and 
outputted to the electric picture signal which corresponds the inputted body image. Drive control of the image pck-u 
element 100 is performed by the image pck-up element driver 200. The electric picture signal outputted from the 
image pck-up element 100 is changed into a digital signal by A/D converter 300, and is recorded on memory 400. 
Record control to this memory 400 is performed by the memory controller 500. By performing predetermined 
interpolation processing described below using the observation picture signal recorded on memory 400, a processor 
600 generates a perfect digital image without lack of each primary signal of R, G, and B, and records this digital 
image on memory 400 again. In the case of this interpolation processing, the pixel arrangement information on a 
mosaic color filter and interpolation coefficient train information in the image pck-up element 100 currently recorde 
on ROM700 are used. In addition, memory 400, the processor 600, and ROM700 are connected through the bus 800 
Thus, it is generated, and the RGB color picture recorded on memory 400 is sent to communication / display device 
an image processing system, etc., and is used for a predetermined use. 

[0014] Drawing 2 shows the example of composition of a mosaic color filter. The mosaic color filter in this operatio 
form consists of three-primary-colors filters of R, G, and B, and the pixel of a same color filter is spatially arranged 
random, as shown in drawing 2 according to the Poisson distribution described below. The color arrangement in a 
mosaic color filter may be arranged here so that it may become random per block picture, and you may decide color 
arrangement suitably to become random by the whole picture regardless of a block. Capacity of the color arrangeme 
information which should be recorded on ROM700, and interpolation coefficient train information can be lessened b 
carrying out color arrangement to random composition within a block, and constituting the whole picture so that the 
block of such composition may be repeated. 

[0015] Drawing 3 is drawing showing the detailed composition of memory 400 and a processor 600. The optical on 
the-strength value detected by each light-receiving pixel of the image pck-up element 100 shown in drawing 1 is 
sequentially read by control of the image pck-up element driver 200, and is recorded on the memory 401 in memory 
400 through A/D converter 300. By constituting the pixel of memory 401 so that it may correspond to the light- 
receiving pixel of the image pck-up element 100, and 1 to 1, the optical on-the-strength detection value from each 
light-receiving pixel of the image pck-up element 100 is recorded on each pixel of memory 401 as a concentration 
value. 

[0016] Based on the color arrangement information on the mosaic filter currently recorded on ROM700, for every 
light-receiving pixel of the image pck-up element 100 first equipped with R filter, a concentration value is read from 
memory 401 and the picture recorded on memory 401 is transmitted to a processor 600. A processor 600 consists of 
CPU601 and DSP602 prepared in order to perform an alignment operation at high speed. The picture signal train re 
from memory 401 and the interpolation coefficient train information currently recorded on ROM700 are sent to 
DSP602, an alignment operation, i.e., a sum-of-products operation, is performed, and the result is recorded on the 
memory 402 for R pictures in memory 400. Such processing is repeated by G and B picture and perfect R which 
finally does not have lack, G, and B primary color picture are recorded on R, G, and the memory 402,403,404 for B 
pictures. Motion control of the process described above until it carried out interpolation generation of perfect R, G, 
and the B picture from the picture signal currently recorded on memory 401 and recorded on the memory 402-404 f 
pictures is carried out by CPU601. 

[0017] A same color pixel value is read in the above-mentioned process, and process in which interpolation 
processing by the alignment operation (sum-of-products operation) is performed is performed per block, as shown 
below. The conceptual diagram of interpolation processing of such a block unit is shown in drawing 4 . Each primar 
color picture of R, G, and B is divided into a block picture as shown in drawing 4 (A), and a sum-of-products 
operation is performed between the concentration values of all observation pixels and interpolation coefficient train 
which are included in each block picture. From the block picture containing a lack pixel as shown in drawing 4 (B) 
according to this interpolation operation, the perfect primary color picture which does not have lack as shown in 
drawing 4 (C) is generated. 

[0018] Drawing 5 is a conceptual diagram in the case of setting up the picture field of near centering on the lack pix 
by which interpolation generation is carried out as a interpolation operation object domain as a modification of 



interpolation processing. With this operation form, in case interpolation generation of the object lack pixel as shown 
in drawing 5 (A) is carried out, the observation pixel concentration value of near corresponding to the lack pixel and 
the interpolation system sequence of numbers corresponding to the lack pixel currently recorded on ROM700 are re 
using the color array information currently recorded on ROM700, and a sum-of-products operation is performed 
using them. By repeating such operation to all lack pixels, as shown in drawing 5 (B), interpolation generation of th 
perfect primary color picture is carried out. 

[0019] An operation of the 1st operation form described above below is explained theoretically. First, imaging ** b 

the veneer color CCD camera at the time of using a RGB filter for a mosaic color filter is formulized with the 

following relational expression. 

[0020] 

g=Hf(l) 

(1) When it rewrites by showing a formula to the content of a matrix and a vector, it is expressed like (2) formulas. 
X)021] 



Equation 



(2) 



[0022] However, f=[fR, fG, and fB] t An ideal RGB color picture (ideal picture) without lack, and g=[gR, gG, and 
gB] t It considers as the color picture of observed RGB. In addition, let fi and gi (i=R, G, B) to f, and let the vector o 
the size of nxl, and g be the vectors of the size of 3nxl, Moreover, the matrix Hi (i=R, G, B) of the nxn size showin 
arrangement of each band filter is defined as follows. 
[0023] 
[Equation 2] 
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[0024] however, matrix Hi ej t the row vector element of an except » Ot it is . That is, matrix Hi It is Ot about the row 
vector which transmits a lack pixel from the unit matrix of nxn. It is equivalent to the replaced thing. Moreover, the 
number of the pixels equipped with R, G, and B filter is respectively set to mR, mG, mB, however mR+mGH-mB =n 
According to these definitions, the relation expressed with the following (4) and (5) formulas is materialized. 
[0025] 
[Equation 3] 
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[0026] Interpolation operation of burying a lack pixel for every primary color picture under the above definition is 

explained. When a interpolation matrix is set to P, the interpolated band picture, i.e., presumed band picture f A , is 

expressed with the following (6) formulas. In addition, in order to simplify expression, the suffix (R, G, B) showing 

band shall omit, and the following formulas shall be defined about arbitrary one out of each three primary color 

pictures. 

[0027] 

[Equation 4] 
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[0028] The interpolation matrix P can be beforehand set up from the color array of a mosaic color filter, and the 
statistical property transcendentally predicted from two or more pictures about a specific picture object, and should 
just write the interpolation matrix P searched for from them in ROM700. 

[0029] In case the interpolation matrix P is set as below, the pixel array of a same color filter explains the operation 
effect in the case of being random, as shown in drawing 2 . (9) A formula shows depending for the precision of the 
interpolation matrix P on the precision of the inverse matrix of the autocorrelation Rgg of Vector g. Although an 
inverse matrix does not exist in accuracy since the rank of Autocorrelation Rgg is m (<n) at the maximum, Rgg can 
think within a rank that they are conditions with the optimal bird clapper to a unit matrix. One of such the condition 
is that arrangement of the lack pixel of Vector g is near at random. Such conditions are explained below using the 
definition of Poisson distribution. 

[0030] Here, a regular probability set which is expressed with the square wave from which an amplitude changes at 
an irregular interval as shown in drawing 6 is considered. The oscillation of a function is Em at an irregular interval 
Shell - Em Or -Em Shell Em It jumps crossing 0. At this time, the course to which a number of 0 (zero intersection) 
of probability distributions which appear in Section t are expressed with (12) formulas is called Poisson distribution 
[0031] 
[Equation 5] 

P. (n ; t) < e Kr (12) 

* n ! 

[0032] However, Pxi (n;t) is the probability used as several n to which the number xi of 0 in Section t was given by 
one observation, and k makes it the average of the number of 0 per unit time. A random pixel array is defined as a 
random unit-impulse train. Then, the unit-impulse train of only a positive value based on Poisson distribution as 
shown in drawing 7 is considered. Rgg is the autocorrelation matrix HHt concerning Matrix H like the following (13 
formulas since Rff will become a unit matrix supposing the ideal picture f is a white property statistically. It become 

[0033] 

Rgg=HRffHt =HHt It is here and is Rff=L (13) 

Then, (15) formulas will be obtained if it asks for the autocorrelation of the positive-value unit-impulse train based 
Poisson distribution from e the definition formula of the autocorrelation Cgg expressed with (14) formulas. 
[0034] 
[Equation 6] 
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[0035] In addition, u (tau) is taken as a certain unit-impulse function which has an impulse by tau= 0. (15) If a 
formula is illustrated, it will become like drawing 8 . (15) Drawing 8 to a formula and autocorrelation matrix HHt A 
analogy predicts having a property like the following formula. 
[0036] 
[Equation 7] 
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[0037] Then, autocorrelation matrix HHt A definition is anew given like the following formula. 
HHt = I+gamma2 W(17) 

(15) It is gamma 2 when it takes into consideration from a formula that the height of the autocorrelation in tau= 0 is 
infinite. A bird clapper is predicted by the small value compared with 1. Then, autocorrelation matrix HHt A false 
inverse matrix is defined like the following formula. 
[0038] 
[Equation 8] 

(HH t )~ = I-t 2 W (18) 
(HH* ) " (HH* ) - (T - 7 2 W)(I + r 2 W) = I - r 4 W ^ £ 



a© 

[0039] (18) Although it is materialized only when the false inverse matrix defined by the formula has the white 
statistical property of the ideal picture f, you may define (18) formulas as an inverse matrix of Autocorrelation Rgg 
it is practical. Although Autocorrelation Rgg will serve as a matrix very near a unit matrix by the above explanation 
same color pixel arrangement of a mosaic color filter is assumed to be a positive-value unit-impulse train based on 
Poisson distribution so that clearly, it can be said that this is the conditions which were most suitable for asking for 
inverse matrix. It is because the matrix itself can give a definition as an inverse matrix as it is when a matrix is a uni 
matrix. It is the autocorrelation matrix HHt in that case that it is the regular array by which a down sampling is carri 
out at the interval decided as a pixel array showed drawing 9 (A). As shown in drawing 9 (B), it becomes a thing 
based on a periodic function, and it stops being suitable for asking for an inverse matrix with a sufficient precision. 
[0040] With the 1st operation gestalt, as shown in (6) formulas, interpolation processing is performed by the 
alignment operation of the view concentration value of the observation pixel g, i.e., an observation pixel, and the 
interpolation matrix P. If a block picture field is made into an NxN pixel, interpolation processing can be defined by 
fitting the above formulization to the vector of the n=NxN element which puts the pixel of the field in order in one 
dimension, and is defined. In addition, although defined as (6) formulas carrying out interpolation generation of the 
pixel perfect picture field from the n-pixel picture field containing both the lack pixel and the observation pixel, a 
interpolation matrix may be defined as generating the lack pixel of an n-m individual from m observation pictures in 



picture field (n-m) in xm procession, and the operation is as having described above. Moreover, when carrying out 
interpolation presumption of the lack pixel from the observation pixel in the near field a center [ a certain lack pixel 
observe ], (6) formulas are redefined like the following (20) formulas. 
[0041] 
[Equation 9] 



C20) 



[0042] In addition, suffix s shows the s-th lack pixel, and is Vector ps and gs. The interpolation and the observation 
vector to the aforementioned near field are expressed, respectively. When defined as carrying out interpolation 
generation of the lack pixel for which it should ask from the n-pixel picture field containing both the lack pixel in th 
aforementioned near picture field, and the observation pixel, they are Vector ps and gs. It consists of both n element 
and when generating a lack pixel from m observation pictures in the aforementioned near picture field, they consist 
m elements. Even in such a case, the above-mentioned operation fundamentally shown to the interpolation matrix P 
explained similarly. 

[0043] According to the above-mentioned 1st operation gestalt, the color picture excellent in color-reproduction 
nature is generable by using the mosaic color filter by RGB. By performing interpolation presumption of a lack pixe 
using the interpolation matrix based on the least square method, a perfect primary color picture without lack can be 
presumed in the state with few errors, and moreover, an operation is performed by the simple sum-of-products 
operation, and is easy to accelerate. A interpolation operation can perform efficient interpolation processing by 
performing to a small picture field by excepting the thin observation pixel of causal relation from interpolation 
processing to the pixel which can cut down the amount of operations upwards and should carry out interpolation 
generation. Moreover, by arranging spatially the pixel of the same color filter in a mosaic color filter at random so 
that it may be based on Poisson distribution, a interpolation matrix can be searched for with a sufficient precision. 
[0044] The 2nd operation gestalt of this invention is explained below. The 2nd operation gestalt of this invention is 
the example which used the complementary color filter for the mosaic color filter of the image pck-up element 100 
the 1st operation gestalt. Here, a complementary color filter means the filter which has the spectral characteristic 
which can pass at least two colors among the three primary colors of light. Since the composition of those other than 
the mosaic color filter in this operation gestalt is the same as that of the 1st operation gestalt, illustration is omitted. 
[0045] An operation of the 2nd operation gestalt is explained theoretically below. With this operation gestalt, 
cyanogen (C), a Magenta (M), and a yellow (Y) complementary color filter shall be used for a mosaic color filter. A 
cyanogen (C) filter inputs simultaneously green (G) and a red (R) color band field, and green (G) and blue (B), and 
Magenta (M) filter are set [ a field ] up for red (R) and blue (B), and a yellow (Y) filter. (Following 21) and followin 
(22) formulas define a picture observation system with these complementary color filters. 
[0046] 

[Equation 10] 
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[0047] However, interpolation matrix PC If the deriving method shown in the 1st operation gestalt is followed, it wi 



ask like the following formula. 
PC =RfCgCRgCgC-l (25) 

However, vector gC The autocorrelation matrix RgCgC is developed like the following formula. 
[0048] 

[Equation 1 1] 
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[0049] 

[Equation 12] 
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[0050] As described above, in this operation gestalt, interpolation generation of the RGB color picture without lack 
carried out by performing the alignment operation defined by (30) formulas from the observation picture inputted by 
the image pck-up element equipped with the mosaic color filter with a complementary color filter. (30) In order to 
calculate a formula, they are Matrices PC, PM, and PY. Although there is the need of recording on ROM700 
beforehand, according to the (26) formulas, they are called for by presuming the cross-correlation between primary 
color pictures transcendentally with the autocorrelation of R, G, and B primary color picture. 
[0051] Although the above explanation showed the example which used the complementary color filter of three 
colors of C, M, and Y, a mosaic color filter may consist of combination of G, C, M, Y, or G, C and Y so that many 
pictures which participate in brightness information greatly may be inputted. Moreover, you may adopt the 
combination which added primary color in order to raise both brightness and sensitivity, and added white (W), 
however W=R+G+B to the complementary color. Also in this case, interpolation generation of the RGB color pictu 
without lack is carried out by one alignment process from an observation picture like the above-mentioned method. 
[0052] Since the wavelength field covered with one filter by using a complementary color filter for a mosaic color 
filter will be extended to the double precision in the case of a RGB filter according to the above-mentioned 2nd 
operation gestalt, sensitivity can be raised, and the good color picture of S/N can be inputted and generated. 
Moreover, by using each primary color picture and the correlation information between primary color pictures whic 
are beforehand prepared in a interpolation means, the optimal conditions can be set up to the target picture, using 
more point and interpolation generation of the color picture near an ideal picture can be carried out. 

[0053 ] The 3rd operation gestalt of this invention is explained below, the 3rd operation gestalt of this invention - an 
image pck-up element -- CMD (Charge Modulation Device) etc. ~ the device in which random access is possible an 
destructive read of the charge moreover accumulated at the light-receiving pixel is possible is used to the pixel in a 



light-recdving side like The whole 3rd operation gestalt block diagram is shown in drawing 10 . An objective image 
is inputted into the image pck-up element 1000 which consists of CMD image sensors by which the image pck-up 
side was equipped with the mosaic color filter first in this drawing. Drive control of the image pck-up element 1000 
performed by the image pck-up element driver 1 100. The color arrangement information on a mosaic color filter is 
recorded on ROM 1200. Read-out control of the picture signal from the image pck-up element 1000 by the image pc 
up element driver 1 100 is performed based on the color arrangement information. The electric picture signal outputt 
from the image pck-up element 1000 is changed into a digital signal by A/D converter 1300, and is held temporarily 
at buffer memory 1400. 

[0054] Moreover, although the code for making the kind of mosaic color filter, a filter color, and the number of a 
block distinguish is recorded as color arrangement information recorded on ROM1200, this code is read from the 
image pck-up element 1000, and the picture signal train recorded on buffer memory 1400 is simultaneously recorde 
on buffer memory 1400 as a header signal which shows whether it is what was observed by what color array. When 
the color array code signal recorded on buffer memory 1400 is decoded by the processor 1500, from ROM1600, a 
predetermined interpolation system sequence of numbers is read, and predetermined interpolation processing is 
performed in a processor 1500 between the observation picture signal trains recorded on buffer memory 1400. 
[0055] It is generated by the above composition as a perfect digital image without lack of each primary signal of R, 
and B, and is recorded on memory 1700. In addition, a processor 1 500 consists of a CPU which controls the whole 
interpolation processing, and DSP which performs interpolation processing by the sum-of-products operation at hig 
speed like the processor 600 in the 1st operation gestalt. Moreover, memory 1700 consists of three memory block fo 
recording each primary signal of R, G, and B. The data transfer between buffer memory 1400, a processor 1500, 
ROM1600, and memory 1700 is performed through a bus 1800. Like the 1st operation gestalt, the RGB color pictur 
finally recorded on memory 1700 is sent to communication / display device, an image processing system, etc., and i 
used for a predetermined use. 

[0056] Drawing 11 is a conceptual diagram for explaining read-out drive control of CMD image sensors. Based on 
the horizontal and perpendicular address signal which are sent from the image pck-up element driver 1 100, the 
amount of optical charges accumulated at each light-receiving pixels 501 l-501n, 5021-502n, 503 l-503n, and 50ml 
50mn is read, when both the signals of the horizontal scanning circuit 20 and the vertical-scanning circuit 30 are ON 
However, since the charge of each light -receiving pixel has the destructive-read function in which it is held, after 
read-out can use the image pck-up element itself as memory. 

[0057] In addition, with the 3rd operation gestalt, an image pck-up system and an image-processing system can be 
separated, and it can be made the composition which can exchange an image pck-up system. In this case, an image 
pck-up system consists of the image pck-up element 1000, the image pck-up element driver 1 100, ROM1200, A/D 
converter 1300, and buffer memory 1400 as a bus interface, and an image-processing system consists of a processor 
1500, ROM1600, memory 1700, and a bus 1800. When using the image pck-up element 1000 equipped with a 
different mosaic color filter, ROM 1200 on which the color arrangement information and code corresponding to it 
were recorded is used. On the other hand, by the image-processing system, all the interpolation system sequence of 
numberses corresponding to the image pck-up system assumed are recorded on ROM 1600, and predetermined 
interpolation processing is performed according to the code signal sent from an image pck-up system. 
[0058] According to the above-mentioned 3rd operation gestalt, by using the image pck-up element in which random 
and destructive read are possible, an image pck-up element can be used as a device which has a memory 
simultaneously with an image pck-up function, and the capacity of the buffer memory 1400 as digital image memor 
can be saved. Moreover, by forming ROM 1200 which recorded the code which shows the color arrangement 
information on a mosaic color filter on the image pck-up system, the composition which can exchange an image pck 
up system is easily realizable. 

[0059] The 4th operation gestalt of this invention is explained below. With the 4th operation gestalt of this invention 
an image pck-up system is constituted from an electronic camera, the picturized image information is recorded on a 
memory medium, and a color picture is reproduced based on the image information transmitted from a memory 
medium in the image-processing system mainly constituted in a personal computer. 

[0060] Drawing 12 is drawing showing the composition of an electronic camera 2000, and this electronic camera is 
equipped with the memory card 3000 which consists of a semiconductor IC, optical memory, magnetic memory, etc 
The image pck-up element 2010 equipped with the mosaic color filter, the image pck-up element driver 2020 and 
ROM2030, and A/D converter 2040 are constituted like the 3rd operation gestalt by the electronic camera 2000. 
However, the image sensors used for the image pck-up element 2010 may be any of the device of CCD, CMD, or 
others, and the image pck-up element driver 2020 should just be constituted to compensate for use of the image 
sensors used. The picture signal inputted by the image pck-up element 2010 is sequentially read by drive control of 
the image pck-up element driver 2020, is changed into a digital signal by A/D converter 2040, and is sent to card 



write-in equipment 2050. The code signal which expresses the kind of color filter with card write-in equipment 205 
from ROM2030 simultaneously is sent. Card write-in equipment 2050 records these observation picture and a color 
filter code on memory card 3000. 

[0061] Drawing 13 is drawing showing the composition of a personal computer 4000, and this personal computer 
4000 is equipped with the memory card 3000 on which the above-mentioned observation picture and the color filter 
code were recorded. In a personal computer 4000, the information currently recorded on memory card 3000 is read 
the card reader 4010. A color filter code is decoded by CPU4020 among them, and an observation picture is recorde 
on memory 4030. Next, based on the decoded color filter information, the predetermined interpolation system 
sequence of numbers currently recorded on the hard disk 4050 by control of the hard disk driver 4040 is read, and 
predetermined interpolation processing is performed in DSP4060 between the observation picture signal trains 
currently recorded on memory 4030. In addition, DSP4060 is constituted by the processor by which a sum-of- 
products operation is performed at high speed. Moreover, you may constitute a interpolation system sequence of 
numbers so that it may record on ROM instead of a hard disk 4050. 

[0062] In addition, although operation machines, such as a color monitor, a keyboard, and a mouse, are connected to 
personal computer 4000, it omits drawing. It is generated by the above composition as a perfect digital image witho 
lack of each primary signal of R, G, and B, and is recorded on the video memory 4070. The RGB color picture 
outputted from the video memory 4070 is changed into a predetermined analog video signal by D/A converter 4080 
and is displayed on a color monitor. In addition, the card reader 4010 in a personal computer 4000, CPU4020, 
memory 4030, the hard disk driver 4040, DSP4060, and the video memory 4070 are connected through the bus 409 
[0063] In the composition explained above, first, using an electronic camera 2000, an electronic camera 2000 can 
picturize the target body under all usable environment, and can record an operator on memory card. They can be rea 
using a personal computer 4000 and it can display as a color picture on a color monitor. In addition, you may 
constitute from a personal computer 4000 so that an operator can apply to various image processings and computer 
graphics using the picturized color picture. 

[0064] According to the above-mentioned 4th operation gestalt, the environment where an image processing with th 
image pck-up operation and the personal computer using the electronic camera and display operation can be 
independently performed to an operator can be offered. That the picture of one sheet recorded on memory card in th 
case should just consist of numbers of pixels equivalent to the number of light-receiving pixels of an image pck-up 
element, since the color arrangement information on a mosaic color filter is coded and recorded, there may be little 
capacity required of memory card. Therefore, an operator can record many pictures by the memory card of one shee 
And in a personal computer, since optimal interpolation processing based on this invention is performed, a high 
definition color picture is reproduced. 

[0065] The 5th operation gestalt of this invention is explained below. The 5th operation gestalt of this invention add 
a **** function to an image pck-up system, and is related with the method of generating the high definition color 
picture of one sheet from two or more pictures inputted continuously. The block diagram of the 5th operation gestal 
is shown in drawing 14 . The picture of a maximum of N sheets as which the image pck-up element 5100, the image 
pck-up element driver 5200, AID converter 5300, memory 5400, a memory controller 5500, a processor 5600, 
ROMS 700, and N memory 5401-540N that records an observation picture in memory 5400 although it consists of 
buses 5800 further were formed, and was inputted from the image pck-up system is recorded by the composition of 
this operation gestalt like the 1st operation gestalt. In memory 5400, the memory 5411-5413 which records R, G, an 
B picture other than these, and the memory 5414 used as a work area in the middle of processing are formed. 
Moreover, in a processor 5600, the correlation arithmetic unit 5630 other than CPU5610 and DSP5620 is formed, th 
correlation operation between the observation pictures recorded on Memory 5401-540N is performed, and the same 
photographic subject's relative location is called for. 

[0066] In addition, you may substitute DSP5620 for the correlation arithmetic unit 5630 by performing a correlation 
operation by the sum-of-products operation. Next, the observation picture by the same color filter between the 
observation pictures recorded on Memory 5401-540N is compounded using the relative location between the 
observation pictures searched for in the correlation arithmetic unit 5630, and it is recorded on memory 5414. 
Although the synthetic picture recorded on memory 5414 turns into a picture by which random arrangement of the 
pixel was newly carried out, the pixel arrangement is called for by CPU5610 using the relative location between the 
observation pictures searched for in the color arrangement information and the correlation arithmetic unit 5630 of a 
mosaic color filter of the image pck-up element 5100 which are recorded on ROMS 700. In CPU5610, further, from 
pixel arrangement of a synthetic picture, a interpolation coefficient train is computed and the sum-of-products 
operation of a synthetic picture and a interpolation coefficient train is performed for every block in DSP5620. The 
perfect primary color picture which makes it such and finally does not have a lack pixel in memory 541 1-5413 is 
recorded. 



[0067] Bfelow, an operation of the 5th operation gestalt is explained. This equipment has a **** function, and by the 
short predetermined time interval, two or more pictures of a maximum of N sheets continue at the time of an image 
pck-up, and it is inputted at it. In addition, each picture by which the **** input was carried out presupposes that it 
picturized while a photographic subject's position shifts relatively to the image pck-up element 5100 by the hand 
which has a camera at the time of photography blurring including the same photographic subject, or moving slightly 
without a photographic subject hardly changing a form. Two or more pictures inputted on such conditions are 
compounded for every observation picture by the same color filter. The conceptual diagram of the picture 
composition is shown in drawing 15 . each observation picture -- setting - one kind of color picture - **** picture 
No. of drawing as shown in 1 and 2, a pixel is random and is picturized in the state where it has been arranged at 
It asks for the same photographic subject's relative location contained in these **** pictures, and by the synthetic 
picture acquired by adding them after performing alignment, as shown in drawing, the density of an observation pix 
improves. Thus, interpolation generation of each primary color picture is carried out by the method described above 
from the acquired synthetic picture. From two or more pictures by which the **** input was carried out with veneer 
color image pck-up equipment from having described above, the color picture of one sheet excellent in resolution is 
generable. 

[0068] Next, when applying to the 5th operation gestalt, it explains that the mosaic color filter arranged at random i 
effective. It considers carrying out the **** input of the observation picture of two sheets with one kind of color 
filter. The relative location of the same photographic subject reflected in the observation picture of these two sheets 
called for according to a correlation operation, and assumes that alignment was already performed. It is the syntheti 
picture acquired by adding them g= [gl t g2 t] t If it carries out, a picture input system will be formulized like the 
following formula. 
[0069] 

[Equation 13] 

g-Hf (31) 



[ K, ] [ H 9 _ 



(32) 



[0070] However, Hi (1 i= 2) is taken as the filter arrangement function matrix corresponding to the i-th picture inpu 
For example, HI Supposing it is a matrix showing original filter pixel arrangement, H2 will be considered to be a 
matrix showing the state where arrangement was spatially shifted by alignment. Next, from the observation picture 
it divides into two processes of relocation within the picture of the observation pixel g, and interpolation presumptio 
and the process which presumes the perfect ideal picture f is defined as them, as shown in the following formula. 
[0071] 

[Equation 14] 

f = Pra = PH V g (33) 

m = H v g (34) 

H v = H* (HH* ) V (35) 
(3 5) ttmZ&BffiffiSn* (HH t ) v *H'<* 0 ffWHH* 
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[0072] (36) Set at a ceremony and it is HI . And H2 If a row vector is rearranged so that unit row vector eijt (i= 1, 



2;j=l ? - m) may come to the first m train respectively, (36) formulas will be rewritten like the following formula. 
[0073] 

[Equation 15] 
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[0074] (37) Non-diagonal element matrix Ik in a formula Rank k is a matrix HL H2 It is dependent on the grade of 
the overlap of an observation pixel. That is, matrix HI H2 Supposing it is defined as observing by completely 
different pixel, it is set to Ik =0 (k= 0), and it can be said that they are the best conditions. In this case, autocorrelatio 
matrix HHt A rank is set to 2m. The on the other hand worst conditions are a matrix HI . H2 It is set to Ik =Im (k=m 
by the case where it is defined as having been observed by the completely same pixel, and is the autocorrelation 
matrix HHt. A rank is set to m. It is thought that it will generally be in these middle states. The ideal matrix H is 
considered according to such analysis. Although in the case of what Matrix H depends on regular arrangement best 
conditions will be acquired if the shift of a picture comes to the valley of a pixel period exactly, it becomes the wors 
conditions when it synchronizes with a pixel period conversely. On the other hand, it will be Ik if pixel arrangement 
made random. It is convenient, when receive, and the amount of position gaps of a picture is not approached, but a 
fixed rank can be expected and equipment is constituted. 

[0075] Interpolation generation of the perfect primary color picture which does not have the composition and lack o 
the picture of one sheet from a **** input picture according to the process which constitutes color arrangement of a 
mosaic color filter from this operation gestalt at random for the above reasons, and is expressed with (33) formulas 
performed. In addition, by the synthetic picture, new random pixel arrangement is realized based on alignment, or th 
interpolation matrix corresponding to it is searched for as follows. 

[0076] Pixel arrangement of a synthetic picture is first computed using the relative location between the observation 
pictures searched for in the color arrangement information and the correlation arithmetic unit 5630 of a mosaic colo 
filter of the image pck-up element 5 100 which are recorded on ROM5700. Next, what is necessary is for the false 
inverse matrix drawn when it is assumed that pixel arrangement which is defined by (18) formulas shown, for 
example in the 1st operation gestalt follows Poisson distribution just to define the optimal interpolation matrix. In 
such a case, it is not necessary to use own transcendental correlation information of a picture, and can ask for a 
interpolation system sequence of numbers only from the pixel arrangement information on a synthetic picture. 
[0077] According to the above-mentioned 5th operation gestalt, when pixel arrangement of the same color filter in a 
mosaic color filter is distributed over **, the resolution of the color picture finally generated can be raised by 
compensating between **** pictures. If color arrangement of a mosaic color filter is made random, improvement in 
resolution can be made to realize efficiently, without being dependent on the relative location of the photographic 
subject between **** pictures. 

[0078] Invention which has the following composition is included in the above-mentioned concrete operation gestal 
The image pck-up element which has arranged spatially at random the mosaic color filter which has two or more 
colors so that the filter position of each color may correspond to an image pck-up pixel, (Composition 1) A storing 
means to store the filter color arrangement information on the aforementioned mosaic color filter, Color picture ima 
pck-up equipment characterized by providing a interpolation means to interpolate the image information for every 
color including lack by interpolation processing in the state where there is no lack pixel, using the image informatio 
outputted from the aforementioned image pck-up element, and the filter color arrangement information within the 
aforementioned storing means. 

(Operation gestalt of corresponding invention) Invention given in composition 1 corresponds to the above-mentione 
1st, 2, 3, and 4 operation gestalt. 



[Translation done ] 



* NOTICES * 



Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original precisely. 
2 **** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



CLAIMS 
[Claim(s)] 

[Claim 1] Color picture image pck-up equipment characterized by providing the following. The image pck-up 
element which has arranged spatially at random the mosaic color filter which has two or more colors so that the filte 
position of each color may correspond to an image pck-up pixel. A storing means to store the filter color arrangeme 
information on the aforementioned mosaic color filter. Image information outputted from the aforementioned image 
pck-up element. A interpolation means to interpolate the image information for every color including lack by 
interpolation processing in the state where there is no lack pixel, using the filter color arrangement information with 
the aforementioned storing means. 

[Claim 2] The color picture image pck-up element in the color picture image pck-up equipment which is characteriz 
by providing the following and which interpolates the picturized color picture information by the interpolation mean 
and restores the original picture. The image pck-up element which has arranged spatially the mosaic color filter whi 
has two or more colors by the batch to interpolate so that the filter position of each color may correspond to an imag 
pck-up pixel at random. A storing means to store the filter color arrangement information on the aforementioned 
mosaic color filter. 

[Claim 3] Color picture image pck-up equipment characterized by providing the following. The image pck-up 
element which has arranged spatially at random the mosaic color filter which has two or more colors so that the filte 
position of each color may correspond to an image pck-up pixel. A storing means to store the filter color arrangeme 
information on the aforementioned mosaic color filter. A storage means to memorize the picture of two or more 
sheets which is outputted by the aforementioned image pck-up element and continues about a time-axis. A position 
gap detection means to detect the same photographic subject's relative location mutually among two or more sheets 
memorized by this storage means, The relative location detected by this position gap detection means is used, the 
picture of two or more sheets memorized by the aforementioned storage means ~ the above — with a picture 
composition means to compound so that the same photographic subject may lap A interpolation means to restore the 
image information for every color including lack to the state where there is no lack pixel, by interpolation processin 
using the image information compounded by this picture composition means and the color arrangement information 
within the aforementioned storing means. 

[Translation done ] 
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